Skip to content

feat(HNT-2888): add Redis key names for crawl state - #21

Draft
mmiermans wants to merge 1 commit into
claude/hnt-2086-message-validationfrom
claude/hnt-2086-redis-keys
Draft

feat(HNT-2888): add Redis key names for crawl state#21
mmiermans wants to merge 1 commit into
claude/hnt-2086-message-validationfrom
claude/hnt-2086-redis-keys

Conversation

@mmiermans

@mmiermans mmiermans commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Goal

HNT-2888

Define the crawl keyspace in one place so the agent and both worker roles derive the same Redis key for a given URL. This slice adds the seven key builders for the crawl keyspace, the page and article fetch timestamps, the lock guards, the page:enqueued and article:enqueued agent markers, and the article:content change-detection key, plus the shared hashUrl; the redis-state client that reads them and the process steps that call them land in later slices.

See the state-store keyspace in docs/ARCHITECTURE.md under review as #14.

Implementation decisions

Decision Approach Why
Derive every crawl key from one module Seven builders plus a shared hashUrl, exported from crawl-common, with no service composing key strings itself The state-store keyspace is shared across the agent and both worker roles, so a single owner keeps their key formats from drifting apart. The namespaces match the state-store table in the Article Crawler tech spec.
Hash the URL with SHA-256 Each builder prefixes its namespace onto the hex digest of the URL Keeps keys a fixed length regardless of URL length, matching the {hash} placeholder the tech spec state-store table uses for every key.
Do not prefix keys with the environment Every builder emits a bare namespace:hash with no dev, stage, or prod segment Each environment has its own Memorystore instance, so the keyspace is already isolated and an environment prefix would add nothing. The tech spec state-store table likewise names its keys without an environment segment.
Trim the URL before hashing, here hashUrl trims surrounding whitespace before digesting So incidental whitespace cannot split one article across two keys. This is the URL normalization deferred from the message-validation boundary, kept in the single place that derives keys so nothing can bypass it.
Add page:enqueued and article:enqueued markers Two agent-only builders beyond the fetch, lock, and content keys the spec lists The agent enqueues a page or live article only when it was not enqueued recently, a marker the tech spec state-store table does not cover but the agent tick needs to avoid re-publishing every minute.

@mmiermans
mmiermans force-pushed the claude/hnt-2086-message-validation branch from 0b762e8 to b59806c Compare July 6, 2026 17:08
@mmiermans
mmiermans force-pushed the claude/hnt-2086-redis-keys branch from 54a1f4b to 2d2de0a Compare July 6, 2026 17:08
@mmiermans mmiermans changed the title feat(HNT-2086): add Redis key names for crawl state feat(HNT-2888): add Redis key names for crawl state Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mmiermans